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Int  roduction 

The  following  chapters  describe  various  aspects  of  lesson  production 
and  use  of  the  PLATO  system  at  Chanute  AFB.  The  first  chapter  ("Aspects 
of  Lesson  Production  at  Chanute")  describes  those  factors  that  were 
most  influential  in  affecting  the  creative  process.  The  second  ("Computer- 
Aided  Management  at  Chanute")  concerns  the  use  of  the  PLATO  system  for 
data  management,  student  testing,  and  other  related  CMI  aspects.  The  ] 

j 

last  chapter  ("A  Survey  of  Chanute  Lesson  Development  Techniques") 
discusses  the  development  of  eight  lessons  in  specific  terms  and  how 
these  lessons  were  modified  to  increase  their  chances  of  validation. 

It  is  hoped  that  taken  together  these  studies  will  provide  some  insight 
into  the  development  and  utilization  of  CBE  materials  at  one  military 
installation. 


Chapter  1:  Aspects  of  Lesson  Production  at  Chanute 


Abstract 


The  purpose  of  this  study  -is  to  examine  the  process  of  lessen 
production  at  Chanute  AFB  during  the  period  of  ISO  |^involveirent . The 
report  considers  four  major  factors  influencing  lesson  production: 
implementation  of  the  ^lean  approach'*  , the  ISD  role  in  lesson  produc- 
tion,  the  transfer  of  programmed  instruction  techniques  to  and  the 

general  method  of  lesson  production  (group  vs.  individual).  Each 
factor  is  discussed  in  terms  of  its  effect  on  the  quantity  and  quality 
of  lesson  production.  This  report  was  submitted  to  the  Chanute  PLATO 
author  staff  for  their  comments  and  suggestions. 


Introduction 


Background  and  Objectives 

This  study  is  intended  to  describe  some  aspects  of  lesson  develop- 
ment at  Chanute,  It  focuses  on  the  process  as  it  unfolded  during  an 
important  period  of  the  project's  existence:  June  1974  to  June  1975. 

This  was  a crucial  phase  of  the  project  when  the  Instructional  System 
Development  (ISD)  group  assumed  direction  of  the  PLATO  effort.  A 
definitive  description  of  the  site  history  is  not  warranted  here. 

(For  further  details,  consult  Himwich,  1977  and  Dallman  et  al . , 1977). 

Only  the  highlights  of  the  process  will  be  covered  in  this  study. 

It  is  hoped  that  the  results  of  this  treatment  will  be  generalizable  to 
other  PLATO  and/or  CBE  installations. 

Method  of  Approach 

Given  the  constraints  as  indicated  above,  the  following  factors  will 
be  discussed: 

1.  implementation  of  the  "lean  approach"  to  lesson  production; 

2.  ISD  involvement  in  lesson  production; 

3.  transfer  of  programmed  instruction  techniques  to  CBE; 

4.  general  method  of  lesson  production. 

Specific  characteristics  of  individual  lessons  have  already  been  discussed 
(Klecka,  1977a,  1977b).  In  the  present  study,  the  factors  that  were 
considered  to  have  the  most  significant  effect  on  lesson  production 
at  Chanute  were  chosen  for  detailed  investigation. 


Implementation  of  the  "Lean  Approach”  to  Lesson  Production 


The  lessons  written  at  Chanute  were  produced  according  to  the 
basic  philosophy  of  the  so-called  "lean  approach".  This  consists  of 
Incorporating  the  bare  essentials  to  cover  the  training  objectives  without 
any  extra  "fat"  that  might  be  construed  as  overteaching  the  student.  The 
emphasis  Is  on  training  personnel.  Consequently,  only  enough  text  Is 
given  to  describe  the  subject  matter  and  only  a few  questions  are  asked 
of  the  student  In  the  course  of  the  lesson. 

In  Chanute's  lessons,  the  student’s  progress  Is  measured  by  a matching 
test  at  the  end  of  the  lesson:  this  Is  known  as  the  Master  Validation 
Exam  (MVE)  which  was  used  originally  for  Insuring  that  90%  of  the  students 
going  through  the  lesson  scored  80%  or  better  on  the  test.  The  test  and 
the  philosophy  of  the  lean  approach  make  a clear  distinction  between  what 
a student  NEEDS  to  know  and  what  Is  NICE  to  know  but  Is  not  ESSENTIAL  to 
the  performance  of  the  task  being  taught.  The  lean  approach,  when 
strictly  applied,  comprises  need-to-know  material  only.  (The  emphasis  Is 
necessary  since  references  to  the  lean  approach  are  used  to  justify  the 
^eclflc  aspects  of  Individual  lessons:  see  Klecka,  1977a,  1977b.) 

The  mission  of  the  Air  Force  technical  training  Is  to  produce  at 
minimum  cost  qualified  Individuals  who  can  effectively  perform  their  duty. 
Thus,  the  students  are  given  Instruction  only  for  their  particular  field 
of  specialization.  Everything  within  this  training  framework  relates  to 
that  specialized  area,  at  least  in  theory.  There  is  no  interest  In  giving 
a broad  range  of  theory  and  practice  In  several  different  areas,  as  is  the 
case  Ir  the  private  sector,  such  as  a university.  This  would  be 


prohibitively  expensive  and  would  serve  no  useful  purpose,  in  the  Air 
Force's  view.  The  airman  is  prepared  to  do  a specific  job,  that  and  no 
other . 

Characteristics 


The  general  characteristics  of  the  lean  approach  can  be  distilled 
as  follows: 

1.  presentation  of  only  material  that  is  essential  for 
performing  a specific  task; 

2.  presentation  of  the  minimum  amount  of  such  essential 
material  to  ensure  adequate  performance; 

3.  testing  the  student  on  a few  important  terms/ components 
that  are  sufficient  to  indicate  familiarity  with  the 
subject  matter  (i.e.,  the  test  samples  the  content 
area  and  is  not  comprehensive) ; 

A.  \^^rlting  the  materials  originally  in  an  overly  lean 
fashion  because  overteaching  is  relatively  more 
difficult  to  detect  than  underteaching;  later  the 
lessons  are  "shored  up"  where  necessary  to  achieve  the 
desired  mastery. 

The  specific  characteristics  of  the  lean  approach  when  applied  to 
CBE  instruction  at  Chanute  include  the  preparation  of  materials  with 
"no  frills"  (Dallman,  1975,  p.  A):  that  is, 

1.  little  branching  except  for  forced  review; 

2.  multiple  choice  questions  and  answers; 

3.  little  feedback  for  correct  or  especially  for  incorrect 
answers ; 

A.  few  elaborate  diagrams  to  accompany  and  elucidate  text; 

5.  little  student  control  of  movement  within  the  lesson; 

6.  sizeable  Instructional  steps,  and 

7.  perceptible  reliance  on  the  techniqc  s of  programmed 
instruction  for  CBE  lesson  construction. 
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These  characteristics  are  significant  since  CBE  lessons  are  comparatively 
expensive  to  produce.  Thus,  including  "frills"  was  expensive  and  of 
doubtful  value,  in  the  Air  Force's  view.  Although  this  philosophy  was 
sometimes  distasteful  to  authors  and  reviewers,  it  was  defended  on  the 
basis  of  cost-effectiveness. 

Discussion 

The  "no  frills"  approach  to  lesson  production  was  implemented  to 
create  efficient  lessons  quickly:  that  is,  lessons  that  presented  the 
material  concisely  and  without  elaborate  diagrams  or  lengthy  discussions. 
However,  some  of  the  lessons  examined  in  the  individual  reviews  had 
elaborate,  slow-to-appear  diagrams.  The  presence  of  such  schematics 
would  seem  to  deviate  from  the  ISD  philosophy;  this  is  especially  true 
since  comprehension  of  the  diagrams  seemed  unnecessary  for  successfully 
completing  the  final  exam. 

The  lean  approach  philosophy  carried  over  to  the  area  of  remediation 
for  incorrect  answers  on  individual  questions  as  well  as  the  end-of-lesson 
test  itself.  In  these  cases,  no  special  help  units  were  constructed 
but  the  student  was  merely  forced  back  through  material  he  had  already 
seen  in  order  to  hopefully  comprehend  it  the  second  time  around.  The 
amount  of  space  required  for  the  lesson  to  be  studied  on-line  would  remain 
the  same  when  this  type  of  remediation  was  used.  Because  Implementation 
of  this  technique  reduced  or  at  least  did  not  worsen  the  computer  storage 
problem,  it  was  a favored  approach.  More  important,  it  did  not  require 
the  creation  of  new  material  and  was  thus  quicker  and  more  efficient  than 
constructing  additional  help  units. 


Examples  of  a deviation  from  the  lean  approach  have  been  documented 
(Klecka,  1977a).  Occasionally,  text  was  presented  which  was  not  tested 
directly  and  thus  did  not  fall  within  the  lesson  objective  of  passing 
the  MVE.  Conversely,  the  lesson  objective,  being  very  broad,  implied  that 
a vast  amount  of  material  would  be  covered.  In  actuality,  only  the 
briefest  introduction  to  the  subject  was  presented.  Thus,  adherence  to 
this  philosophy  of  lesson  construction  was  not  universal. 

Summary 

In  theory,  the  lean  approach  sounds  very  appropriate  for  implementing 
Air  Force  technical  training:  to  give  the  airman  the  minimum  but  adequate 
amount  of  preparation  for  the  actual  field  experience  where  he  will  really 
learn  his  job.  In  practice,  Chanute  authors  found  it  hard  to  follow  that 
approach  rigorously  and  hence  some  aspects  of  the  lesson  materials  varied 
in  emphasis  and  adherence  to  it.  Nevertheless,  scattered  individual 
reports  from  other  bases  (e.g.,  personal  interview)  have  indicated  that 
the  current  level  of  CBE  training  is  probably  adequate  for  basic  prepara- 
tion of  the  airman.  Field  evaluations  indicate  average  or  above  average 
results  (Dallman  et  al.,  1977).  However,  it  does  not  follow  that  this 
was  due  either  to  rigorous  or  relaxed  applications  of  the  widely-publicized 
lean  approach. 
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ISP  Role  in  Lesson  Production 

Initial  Involvement 

At  Chanute  during  the  period  covered  by  this  study  (mid  '74  to 
mid  '75),  the  Instructional  System  Development  (ISD)  group  directed  the 
process  by  which  the  CBE  lessons  would  be  prepared.  This  group  had  primary 
responsibility  for  a developmental  process  known  as  the  "front-end 
analysis".  It  consisted  of  the  following  elements: 

1.  preparation  of  a task  analysis,  a specialized  process 
which  encompasses  describing  and  analyzing  the  job 
tasks  in  order  to  determine  appropriate  training 
(Department  of  the  Air  Force,  30  January  1974) ; 

2.  determination  of  skill  levels  for  performing  the 
required  task(s) ; 

3.  preparation  of  the  lesson  objectives; 

4.  creation  of  criterion  tests  to  check  the  performance 
level  of  the  students. 

The  PLATO  authors  were  not  involved  in  the  preparation  of  the  analyses 
nor  in  the  composition  of  the  component  parts. 

Initially,  the  ISD  staff  believed  that  the  PLATO  author  staff  was 
sufficiently  qualified  and  experienced  that  a further  breakdown  or 
guidelines  for  either  subject  matter  or  instructional  design  would  not 
be  needed.  Indeed,  ISD  estimated  that  materials  could  be  produced  within 
a range  of  50  man-hours/contact-hour  of  instruction  (early  Interview  with 
project  head)  to  100  hours/contact-hour  (Dallman,  1974,  p.  2).  These 
estimates  were  made  despite  the  fact  that  a preliminary  expenditure  of 
445  man-hours/contact-hour  (Green,  1973,  p.  27)  was  required  to  produce  a 
rough  draft  and  650  hours/contact-hour  to  produce  a finished  lesson 
(Main  et  al. , 1973,  p.  38-j ) . Similarly,  it  was  known  that  Aberdeen 


required  slightly  less  than  300  hours  of  effort  for  each  contact  hour  of 
material  produced  (Dare  et  al.,  1975,  p.  85). 

In  the  preliminary  stages  of  the  project,  the  author  staff  may  have 
been  especially  critical  of  the  ISD  prepared  materials  for  two  reasons. 
First,  because  the  authors  had  not  taken  part  in  the  front-end  analysis 
and  because  of  limited  communication  between  the  groups  (e.g. , separate 
offices) , the  authors  proceeded  on  the  basis  of  the  documentation 
furnished  by  the  ISD  staff  without  the  background  and  rationale  behind 
those  documents.  Second,  the  ISD  staff  expected  to  achieve  a dramatic 
speed-up  in  lesson  production  by  eliminating  some  of  the  styles,  habits, 
and  strategies  which  authors  found  interesting  and  motivating,  but  which 
ISD  found  unnecessary. 

The  lesson  objectives  were  also  prepared  in  advance  for  the  authors. 
These  usually  consisted  of  a standard  approach  toward  the  material  in  the 
lesson  (i.e.,  "without  reference,  identify  basic  facts  and  state  general 

operating  principles  of  ") . This  goal  had  to  be  satisfied  by 

passing  an  end-of-lesson  test  with  an  80%  or  higher  score.  The  striking 
similarity  in  the  format  of  the  objectives  within  different  lessons  lies 
in  their  common  origin  with  the  ISD  group  and  the  people  in  charge  of  the 
vehicle  course  itself.  The  authors  were  provided  with  only  the  general 
topic  of  each  lesson.  They  had  few  guidelines  about  the  breadth  and  depth 
of  the  coverage  desired.  Even  the  end-of-lesson  tests  were  an  imperfect 
guide  to  preparing  the  lesson;  they  only  sampled  the  lesson  content. 
Subsequent  Revisions 

The  authors  asserted  that  the  material  presented  to  them  by  ISD  was 
very  incomplete  and  unsatisfactory.  This  was  especially  apparent  in  the 
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I 

lesson  objectives  and  end-of-lesson  tests:  it  was  the  considered 
opinion  of  the  authors  that  these  components  were  not  sufficient  as 
guidelines  for  the  preparation  of  lesson  materials.  In  view  of  these 
complaints,  the  ISD  staff  reversed  its  assumptions  and  took  a rather 
conservative  view  of  the  authors'  subject  matter  expertise  and  instructional 
design  experience.  The  following  major  revisions  were  subsequently  made 
in  the  method  of  operation: 

1.  the  end-of-lesson  tests  (the  master  validation  exams 
or  MVE's)  were  rewritten  by  ISD  or  in  some  cases 
rewritten  by  the  authors  and  approved  by  ISD  or 
vehicle  course  personnel; 

2.  teaching  points,  a more  detailed  outline  of  what  the 
lesson  contained  than  presented  by  the  lesson  objec- 
tive, were  furnished  by  ISD  to  the  author  staff;  and 

3.  subject  matter  experts  (SME's)  were  brought  into  the 
project  to  lend  their  expertise  to  preparation  of 
lesson  materials. 

Thus,  delays  caused  by  insufficient  input  from  ISD  to  the  author  staff 
ranged  from  a few  days  to  a few  weeks.  These  delays  came  at  the  commence- 
ment of  lesson  development,  a crucial  period  for  the  author  staff  (Misselt, 

1975,  p.  6). 

The  ISD  distillation  of  raw  material  was  not  without  its  benefits 
for  the  authors.  They  were  presented  with  some  of  the  material  which  they 
could  shape  and  mold  according  to  their  specific  inclinations.  That  is  to 
say,  the  authors  did  not  have  to  "start  from  scratch"  (always  a painful 
procedure)  but  had  something  to  work  with  right  away  (i.e.,  the  teaching 
points).  Nevertheless,  the  authors  were  more  highly  constrained  in  the 
modified  relationship  to  the  ISD  staff  and  raised  objections  about  certain 
ISD  components  (i.e.,  the  MVE's  mentioned  above).  Overall  however,  the 
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ISD  group  provided  structure  and  organization  in  this  area  of  nascent 
CBE  development . 

There  was  also  some  feedback  from  the  ISD  group  during  and  after 
the  lesson  writing  stage.  The  ISD  personnel  would  suggest  adding  a 
question  here  or  there.  Their  comments  reinforced  the  basic  ISD  strategy 
of  producing  lessons:  lessons  were  to  be  composed  of  narrative  portions 
with  a few  questions  interspersed  somewhat  at  random.  At  the  end  of  the 
lesson  the  student  would  take  a test  to  see  if  he  had  met  the  objective. 
Thus,  the  ISD  group  offered  mainly  minor  additions  or  changes  rather  than 
radical  instructional  strategy  modifications. 

Summary 

It  can  be  seen  that  the  ISD  group  offered  the  authors  a starting 
point  for  the  composition  of  the  materials.  What  emerged  typically  was 
a lesson  whose  outline  (i.e. , the  objectives  and  "teaching  points")  was 
determined  by  the  ISD  personnel  but  with  the  majority  of  its  specific 
aspects  being  created  by  the  author  with  input  from  course  staff. 

Either  because  of  time  constraints  or  because  of  statements  and  suggestions 
from  the  ISD  staff,  endeavors  requiring  initial  time  investment  for  poten- 
tial long-term  payoff  were  not  investigated. 

The  ISD  group  concentrated  on  a low-risk,  low-potential-gain  method 
of  operation.  This  approach  was  probably  justified  by  the  project  objec- 
tives and  the  history  of  the  site.  A knowledge  of  that  history  aids  the 
understanding  of  why  lessons  produced  at  Chanute  are,  in  general,  not 
"flashy"  or  "clever",  but  nondescript  and  methodical.  The  creativity  of 
the  authors  may  have  been  limited  by  these  constraints;  in  addition,  no 
radical  modifications  to  the  basic  ISD  starting  point  were  permitted  due  to 


their  ostensible  cost-consciousness. 
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Transfer  of  Programmed  Instruction  Techniques  to  CBE 

Influence 

The  writing  of  CBE  materials  at  Chanute  followed  the  traditional 
approach  toward  lesson  development:  that  is,  the  pedagogical  strategies 
of  programmed  instruction  (PI).  Specifically,  the  structure  of  the  lessons 
tended  to  be  linear  in  construction  with  little  branching  to  optional  or 
helping  units.  Also,  the  use  of  graphical  displays  was  somewhat  limited, 
consistent  with  the  adoption  of  the  lean  approach  to  production. 

For  the  most  part,  the  individual  lessons  shared  a common  format 
consisting  of  the  following  elements: 

1.  statement  of  the  objective; 

2.  text  portions  with  limited  interaction; 

3.  end-of-lesson  test  (master  validation  exam  or  MVE) . 

As  mentioned  previously,  ISD  influence  was  most  strongly  felt  in  the 
construction  of  the  objectives  and  MVE's;  however,  there  was  little  control 
on  their  part  for  the  textual  portions  of  the  lesson.  The  ISD  staff 
merely  indicated  that  the  text  be  tailored  to  the  subject  matter  of  the 
lessons,  but  it  was  up  to  the  individual  author  to  design  these  sections. 

An  additional  factor  encouraging  the  utilization  of  the  PI  format 
may  have  been  the  limited  knowledge  that  some  of  the  authors  had  in  the 
use  of  various  TUTOR  language  commands.  Fewer  and  simpler  commands 
(i.e.,  -at-,  -write-)  would  be  needed  for  a linear  lesson  with  little 
branching.  Also,  the  limited  use  of  "hints"  after  an  incorrect  answer  in 
the  programmed  instruction  approach  may  have  lent  Itself  readily  to 
Incorporation  within  the  "bare-bones"  framework.  There  was  little  incentive 
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I < to  provide  extensive  remediation  sequences  either  after  an  incorrect 

I response  was  given  or  in  a separate  help  unit  • 

Discussion 

The  above  considerations  have  validity,  particularly  in  terms  of 
. producing  a large  quantity  of  instruction  while  minimizing  costs.  However, 

it  must  be  pointed  out  that  the  PLATO  system  offers  many  capabilities  for 
i presenting  instruction  in  efficient  and  effective  formats  that  have  not 

been  explored  at  Chanute.  Individual  branching  for  remediation  or  extra 
information  has  wide  application  for  improving  the  effectiveness  of  the 
instruction.  Specific  answer  contingencies  for  pre-determined  incorrect 
responses  can  also  be  used  to  provide  the  student  with  appropriate  remedia- 
tion. Immediate  knowledge  of  results  on  end-of-lesson  tests  can  give  the 
student  helpful  feedback  on  his  level  of  proficiency.  Hence,  Chanute 's 
implementation  may  be  described  as  attempting  to  achieve  cost-effectiveness 
by  holding  development  costs  to  a minimum,  rather  than  by  greatly  reducing 
the  training  needed  by  use  of  elaborate  pedagogy  or  student  management 
strategies. 

Effect  on  Attitudes 

Special  Purpose  Vehicle  Maintenance  instructors  were  asked  for  their 
views  on  the  results  of  using  various  forms  of  1)  structional  aids  (i.e., 
visual,  PLATO  lessons,  study  guides/workbooks,  lectures,  laboratory,  and 
programmed  Instruction).  Although  the  survey  was  limited  to  a small 
database  (17  respondents),  it  may  be  Informative  to  consider  their  feelings 
on  PI.  Most  of  the  respondents  disagreed  with  the  statement  "I  get  very 
good  results  using  programmed  texts''^  (average:  3.89  on  a 0-9  very 

^For  data  and  results  of  the  questionnaire,  see  Dallman,  et  al.,  1977. 
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strongly  disagree  - very  strongly  agree  scale) . They  were  non-committal 
on  the  PLATO  lessons  (5.55).  The  standard  deviations  for  the  above  items 
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were  2.14  and  2.15  respectively. 

Summary 

It  is  of  course  obvious  that  few  techniques  such  as  extensive  hints 

I 

or  remediation  sequences  have  been  explored  in  the  Chanute  lessons  even 
i though  little  extra  effort  would  have  been  required.  The  heavy  reliance 

t 

j on  tried-and-true,  even  stereotyped,  training  techniques  precluded  this 

utilization.  The  limited  skill  of  the  authors  with  the  TUTOR  programming 
language  may  have  been  another  factor.  However,  the  restrictive  influence 
of  the  programmed  instruction  format  was  probably  Instrumental  in  the 
limited  employment  of  selected  PLATO  system  capabilities. 
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General  Method  of  Lesson  Production 

A variety  of  organizational  structures  have  been  used  in  PLATO 
courseware  development,  and  many  groups  have  altered  their  structure  in 
keeping  with  shifting  goals  at  various  points  in  their  project's  history 
(Mahler,  et  al.,  1976).  Such  was  also  the  case  at  Chanute.  In  the  early 
phase  of  the  project,  the  authors  worked  independently  on  lessons  in  their 
own  area  of  content  expertise  and  Instructional  experience.  In  this  mode 
there  were  no  guidelines  for  a common  instructional  strategy  and  the 
authors  had  considerable  freedom  in  developing  their  individual  styles. 

Each  had  responsibility  for  content  and  lesson  strategy  as  well  as 
the  actual  TUTOR  coding.  This  mode  of  operation  can  be  considered  the 
"individual  approach". 

Later,  when  the  project  goals  changed  to  encompass  development  of  CBE 
materials  as  part  of  an  extensively  revised  curriculum,  it  became  obvious 
to  the  new  project  director  (the  head  of  the  ISD  staff)  that  a restructuring 
of  the  PLATO  authoring  group  was  required.  In  the  first  place,  it  became 
necessary  for  some  authors  to  develop  materials  for  content  units  that  they 
had  not  taught  previously  and  in  which  they  had  had  no  special  training. 
Hence,  their  subject  matter  expertise  was  not  broed  enough  to  Include 
these  newly  assigned  areas.  Secondly,  the  ISD  leader  and  his  staff  were 
not  comfortable  with  the  instructional  strategies  evolved  by  the  individual 
authors.  This  concern  was  due  in  part  to  the  inexperience  of  the  ISD 
team  with  CBE  as  a medium  and,  more  Importantly,  because  elaborate 
strategies  were  not  required  for  the  limited  objectives  and  fundamental 
nature  of  the  content  planned  for  the  new  curriculum.  The  ISD  group 
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wanted  to  work  with  "proven"  strategies  rather  than  "experimental"  ones. 

To  overcome  the  twin  deficiencies  of  specialized  content  expertise 
and  instructional  design  philosophy  and  training,  the  project  direc- 
tor instituted  a "team"  approach  to  development  of  the  required 
materials.  This  approach  is  described  in  the  following  section.  It 
is  contrasted  with  the  individual  approach,  used  extensively  at  an  early 
stage  of  the  project  and  also  concurrently  with  the  team  approach  in 
some  instances. 

Team  Approach 

For  a major  portion  of  the  project  effort  at  Chanute,  a variety 
of  team  approaches  was  considered  an  optimum  way  to  produce  the  most 
efficient  and  maximum  number  of  effective  CBE  lessons.  Although  several 
authors  had  previously  been  instructors  in  the  vehicle  maintenance 
course  for  the  bulk  of  the  period  during  which  most  of  the  PLATO  lessons 
were  produced,  the  production  managers  for  the  PLATO  lessons  assumed, 
when  it  came  to  curriculum  development,  that  the  authors  themselves  had 
little  subject  matter  expertise  and  less  Instructional  programming 
experience.  An  instructor  may  have  taught  a specific  block  of  instruc- 
tion several  times,  but  he  might  know  very  little  about  the  material 
in  other  blocks.  The  authors  were  all  recent  Instructors,  but  were 
not  experts  in  material  for  all  topics  for  which  they  wrote  lessons. 

In  this  view,  the  main  thing  that  the  authors  could  provide  was  a 
facility,  albeit  a limited  one,  in  the  TUTOR  programming  langugage. 

To  remedy  this  deficiency,  the  following  trl-partite  organization 
was  suggested  for  a typical  production  team. 
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1.  An  Instructional  designer,  ISD  staff  (part-time), 
who  acted  as  a resource  to: 

a.  aid  authors  who  voluntarily  sought  him  out;  and 

b.  review  lessons  and  eliminate  non-essential  infor- 
mation. 

2.  A TUTOR  author  or  programmer,  PLATO  staff  (full-time). 

3.  A subject-matter  specialist  (SMS) , course  staff  (part- 
time)  , who  acted  as  a resource  to: 

a.  provice  content  for  the  programmer;  and 

b.  provide  subject  matter  expertise  for  the  authors. 

In  addition,  other  course  instructors  (not  assigned  to  the  project) 
were  also  consulted  as  well  as  reference  books,  old  manuals,  and  other 
related  materials.  This  organization  was  necessitated  by  the  belief  that 
one  person  alone  could  not  contribute  all  the  expertise  and  experience 
needed  for  effective  and  efficient  lesson  production. 

The  operating  principle  was  that  this  method  would  succeed  in  pro- 
ducing lesson  material  and  in  time  the  organization  could  be  modified. 

The  TUTOR  author  could  gain  some  of  the  instructional  programmer's 
expertise  as  each  could  learn  from  the  other.  In  fact,  it  was  believed 
that  the  author  and  instructional  programmer  could  produce  lessons  inde- 
pendently of  each  other,  but  there  would  always  be  a need  for  an  SMS  to 
assist  both.  There  are  several  subject-matter  areas  within  the  vehicle 
maintenance  group  that  would  necessitate  individual  specialists  in  these 
areas  on  a continuous  basis.  In  addition,  the  various  SMS's  could  not 
learn  the  TUTOR  language  quickly  enough  to  be  efficient  at  lesson  production 
in  their  individual  areas  of  content  expertise. 
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The  lessons  were  reviewed  both  while  in-progress  and  also  after  a 
first  draft  was  completed.  Usually  an  SMS  examined  it  for  accuracy  of 
content,  although  in  an  interview,  a former  staff  member  commented  that 
occasionally  two  SMS’s  would  disagree  on  the  "correctness"  of  a particular 
point.  The  lessons  were  also  reviewed  by  other  members  of  the  group; 

' these  "peer  reviews"  were  viewed  positively  for  their  contribution  to 

I improving  the  materials,  according  to  former  Chanute  authors.  Reviews  by 

"outsiders"  (l.e.,  MTC,  AFHRL)  were  considered  to  have  little  value, 
since  it  was  believed  that  the  external  reviewers  would  not  be  aware  of 
the  problems  and  constraints  under  which  the  authors  had  to  work. 
Independent  Approach 

In  no  sense  were  any  of  the  authors  fully  Independent  of  other 
members  of  the  Chanute  PLATO  project  or  the  vehicle  course  personnel. 
Subject  matter  specialists  and/or  other  staff  members  had  to  be  consulted 
from  time  to  time  for  assistance  in  coding  or  composing  particular  sec- 
tions. However,  as  the  author  gained  familiarity  with  the  subject  matter 

and  the  TUTOR  language,  there  was  a natural  tendency  to  become  less 
reliant  on  others  and  more  dependent  on  oneself.  An  additional  factor 

might  have  been  the  desire  to  increase  the  production  rate  of  the 
lessons  or  even  try  out  more  individualized  approaches  to  lesson  construc- 
tion. 

Summary 

Since  precise  data  on  rates  of  lesson  production  were  not  kept,  it  is 
difficult  to  evaluate  the  success  of  the  group/individual  approach  in 
cost-effectiveness.  However,  according  to  former  authors  at  Chanute  who 
were  interviewed,  the  group  approach  had  some  merit  and  proved  to  be  a 
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viable  mode  of  operation.  In  theory.  It  brought  together  the  Instruc- 
tional designer,  the  computer  programmer,  and  the  subject  matter 
specialist  to  pool  their  resources.  In  practice,  not  all  members  of  the 
team  contributed  equally,  and  thus  one  or  more  members  had  to  work 
somewhat  Independently  In  order  to  produce  a draft  version  of  the  lesson. 
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Conclusions 

^ The  instructional  strategies  and  managerial  techniques  of  the  ISD 

staff  were  successful  for  increasing  the  lesson  production  rate  to  a 
; level  so  that  materials  for  the  entire  vehicle  repair  course  could  be 

i produced  In  the  allotted  time  with  the  existing  staff.  Management  by 

ISD  caused  a restructuring  of  tasks,  procedures,  and  roles  into  a team- 
i oriented  production  staff.  Instructional  design  by  ISD  meant  curtailing 

I 

experimental  approaches  and  adoption  of  standard  strategies  and  techniques. 
By  Introducing  the  lean  approach  and  enforcing  the  rule  of  "no  new 
strategies,  no  new  hardware",  both  a floor  and  a ceiling  were  placed  on 
the  potential  successfulness  of  the  final  product.  Specifically, 
because  familiar,  Pl-like  traditional  teaching  techniques  were  employed, 
the  instructional  materials  had  very  little  probability  of  failing  to 
teach  adequately  and  were  also  very  unlikely  to  produce  sweeping  gains 
in  performance,  retention,  attitude,  or  time  savings. 

Although  departures  from  the  ISD  techniques  and  guidelines  can  be 
found,  in  general  the  materials  follow  this  conservative  philosophy. 

Though  some  decisions  about  lesson  design  seem  to  have  been  made  on  the 
basis  of  expediency  rather  than  ISD  principles,  the  resulting  training 
course  has  nevertheless  met  its  objectives. 
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Chapter  2:  Computer-Aided  Management  at  Chanute 


Abstract 

The  purpose  of  this  study  is  to  examine  one  organization's  efforts 
to  reduce  the  time  and  cost  while  improving  the  effectiveness  of  various 
management  functions  by  using  the  PLATO  IV  computer-based  education 
(CBE)  system.  The  site  selected  for  this  study  was  Chanute  Air  Force 
Base,  whose  PLATO  project  staff  made  extensive  use  of  the  data  collection 
and  retrieval,  student  records,  and  other  available  resources  of  the 
computer  system.  The  data  collected  included  student  test  scores  and 
specific  responses  to  individual  test  questions.  The  need  for  such  data 
collection  and  management  of  student  performance  scores  has  been  previously 
pointed  out.  This  report  was  submitted  in  draft  form  to  Chanute  PLATO 
staff  for  their  comments  and  suggestions. 
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Introduction 


The  purpose  of  this  study  is  to  examine  one  organization's  efforts 
to  reduce  the  time  and  cost  of  various  management  functions  in  a military 
training  environment  by  using  the  PLATO  IV  computer-based  education  (CBE) 
system.  The  site  selected  for  this  study  was  Chanute  Air  Force  Base, 
whose  PLATO  project  staff  made  extensive  use  of  the  data  collection, 
student  records,  and  other  software  resources  of  the  CBE  system.  The 
data  collected  included  student  test  scores  and  specific  responses  to 
individual  questions.  The  need  for  such  data  collection  and  management 
of  student  performance  scores  has  been  previously  pointed  out  (Dallman, 
1975a,  p.  53). 

In  order  to  gather  data  for  the  Service  Test  of  PLATO  IV  at  Chanute, 
the  Training  Research  Applications  Branch  (TRAB)  and  the  instructor  per- 
sonnel in  the  target  courses  administered  pre-  and  post-tests,  recorded 
student  block  grades,  remedial  instruction  time  and  washback  rates  (Dallman 
1974,  p.  4).  Since  the  completion  of  the  data  collection  for  the  Service 
Test,  similar  course  management  activities  have  been  carried  on  by  the 
reorganized  PLATO  authoring  staff  in  cooperation  with  course  instructors. 
The  capabilities  of  the  system  have  been  used  in  order  to  make  data 
collection  and  management  more  efficient  and  to  ease  the  burden  of  the 
instructors  from  time-consuming  tasks. 

The  intent  of  this  report  is  to  describe  the  management  applications 


of  PLATO  at  Chanute.  Chanute* s techniques  are  not  presented  as  an  optimal 
way  to  utilize  all  system  data-related  management  capabilities  but  as  an 
example  of  one  site's  initial  efforts  in  this  area. 


Selected  Capabilities  of  the  PLATO  IV  System 

The  PLATO  IV  CBE  system  has  many  capabilities  for  data  collection  and 
retrieval  that  can  be  employed  by  the  average  user.  Like  any  advanced 
computer  system,  it  can  manipulate  databases  to  provide  information  for 
decision-making  or  record-keeping.  It  is  especially  convenient  to  use 
the  system  to  monitor  or  manipulate  Information  that  is  generated  by 
on-line  student  use  (e.g.,  test  scores,  responses  to  interaction,  accessing 
of  "help"  sequences,  etc.).  The  collected  data  are  available  when  needed 
by  the  instructor  to  monitor  the  student's  progress  and/or  detect  the  need 
for  remediation. 


Figure  1:  MVE  Times  and  Grades 
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Utilization  of  System  Manasement  Capabilities 
Monitoring  Student's  Progress 

Data,  including  scores  on  tests  and  individual  responses  to  test 
items,  have  been  collected  for  students  in  the  Special  Purpose  Vehicle 
course.  In  addition,  data  have  been  collected  for  students  in  the  General 
Purpose  Vehicle  course.  For  the  latter  course,  the  data  were  of  a 
longitudinal  or  chronological  variety  and  also  included  course  completion 
times  and  bi-weekly  tests  known  as  block  exams. 

Chanute  PLATO  author  staff  created  programs  to  collect  and  display 
a variety  of  student  performance  data.  In  the  on-line  student  roster 
(maintained  on  the  PLATO  system),  the  student  is  listed  by  last  name,  and 
the  scores  on  the  master  validation  exams  (MVE's)  are  recorded  along  with 
the  amount  of  time  that  the  student  needed  to  finish  each  test  (Figure  1) . 
The  number  of  minutes  needed  for  completion  of  the  MVE  is  important  since 
it  probably  indicates  the  student's  level  of  knowledge  and  rate  of  progress 
as  well  as  showing  the  instructor  which  students  are  having  difficulty 
with  the  material.  That  is,  a long  completion  time  and  a low  score  may 
indicate  the  need  for  remediation  or  at  least  consultation  with  the 
student  to  ascertain  a possible  problem  with  comprehension,  while  a fast 
completion  time  coupled  with  a high  score  suggests  some  proficiency  in  that 
area. 

While  collecting  the  student  responses  and  grading  the  test,  the 


computer  implements  the  scoring  formula  or  "correction  factor"  in  use  at 
Chanute  (total  score  = [number  correct]  - [1/4  x number  wrong]).  Thus,  the 
mathematical  grading  functions  that  would  have  to  be  performed  manually  by 
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the  instructor  can  now  be  performed  much  more  rapidly  and  accurately  by 
the  computer.  In  addition,  student  data  can  be  stored  for  selective 
retrieval  by  the  instructor  or  student:  the  instructor  can  use  the 
data  to  determine  the  student's  level  of  proficiency  and  the  student  can 
monitor  his  own  progress  and/or  need  for  remediation.  In  addition,  the 
test  itself  may  need  to  be  modified  based  on  the  results  of  student  data 
collection.  That  process  of  test  modification  is  discussed  in  a later 
section. 

Information  Storage  and  Retrieval 

The  data  that  are  collected  are  easily  accessible  to  instructors 
and/or  their  supervisors.  The  options  available  in  the  Chanute 
Information  retrieval  package  are  listed  on  the  index  page  of  the  pro- 
gram (Figure  2).  A permanent  record  of  the  desired  data  can  be  produced 
by  means  of  the  Varlan  copier  (a  device  attached  to  a PLATO  terminal 
to  record  images  from  the  screen  display)  in  the  authors'  office  at 
Chanute.  The  number  of  instructor  and/or  supervisor  comments  relating 
to  questions  on  the  tests  is  listed  (Figure  3) . The  comments  can  also 
be  accessed  by  the  Instructor  or  supervisor  from  the  same  Information 
retrieval  package. 

The  information  required  by  Air  Force  Form  668  for  recording 
test  scores  has  been  placed  on-line  (Figure  4).  The  responses  from 
the  students  are  now  recorded  automatically  after  they  take  a test 
and  the  form  is  filled  in  without  the  Instructor's  intervention.  The 
effort  needed  for  what  was  once  a "time  consuming  clerical  function" 
(Dallman,  1975,  p.53)  has  been  reduced  from  about  an  hour  to 
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Figure  2:  Test  Times  and  Grades 


INDEX 


Choose  one  ^ 

a)  UPDATE 

b)  LESS(M<  TIME  - copy 

c)  MVE  TIME/GRADE  - copy 

d)  BLOCK  TEST  - copy 


Info  on  last  update 

Data  file  name:  spdata 

Date  updated  : 01/01/76 

Time  updated  : 12.43.29 


The  date  is  01/05/76 


The  time  is  15.25.30 


Figure  3:  Instructor/Supervisor  Comments  for  each  Test 


TEST  # 


//  TIMES 
ADMIN 


//  INSTR 


//  SUPV 
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Figure  4 : On-Line  AF  Form  668 

Class  # 760414ad  11  Students 

Test  It  tin 

incorrect  responses 

correct 

abed  answers 

1 3 0 0 0 b 

2 0 0 0 2 c 

3 0 0 0 0 a 

4 0 0 1 0 b 

5 0 0 0 0 d 

6 0 0 0 0 c 

7 0 0 0 0 a 

8 0 0 1 0 d 

9 0 0 0 0 b 

10  0 0 0 0 a 

11  0 1 0 0 d 

12  0 0 0 0 c 

13  1 0 0 0 b 

14  0 0 0 0 d 

15  0 1 0 0 d 

16  0 0 0 0 b 

17  0 0 0 0 c 

18  0 1 0 0 a 

19  0 0 0 0 a 

20  2 8 0 0 d 

-DATA-  for  another  class 
-LAB-  for  composite 
-SHIFT  LAB-  objectives 
-NEXT-  choose  another  test 


10-20  minutes.  Since  approximately  7 tests  are  given  during  the  6-week 
course  by  each  of  the  11  instructors  and  12  of  the  29  tests  included 
in  the  course  have  thus  far  been  put  on-line,  the  time  saved  can  be 
calculated  as  follows: 

7 tests  12  tests  50  min  1 hr  „ , , , . . ~ 

r ; — X — X X — ^ X 11  instructors  = 

6 weeks  29  tests  1 avg  test  60  min 

4.5  instructor-hours/week 

Thus,  a total  of  more  than  4 instructor-hours  per  week  has  been  saved. 

If  all  tests  were  on-line,  a total  savings  of  approximately  9 man-hours 
per  week  could  be  projected.  The  net  savings  in  dollars  is  difficult  to 
calculate  since  it  is  not  known  how  many  hours  were  required  to  produce 
the  original  data  collection  lessons.  Nevertheless,  the  benefits  resulting 
from  the  utilization  of  such  lessons  are  significant  and  widely  applicable. 


The  larger  saving  was  estimated  by  the  instructors,  while  the  more 
conservative  estimate  is  found  in  the  PLATO  IV  Data  Automation  Requirement 
(DAR)  endorsed  by  Major  General  Edwin  W.  Robertson  II,  on  9Aug76.  Depart- 
ment of  the  Air  Froce,  Headquarters  Chanute  Technical  Training  Center  (ATC) , 
Chanute  Air  Force  Base,  Illinois  61868.  Sec.  2.g.  (3). 
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Specific  Applications  of  Data  Management 

Author  Mode;  POl  Editor 

At  one  point  in  the  project,  it  was  believed  that  plans  of  instruction,  j 

(POI's)  outlines  for  teaching  a particular  segment  of  material,  were 
subject  to  a series  of  rapid,  yet  relatively  minor  format  changes.  Although 
the  re-typing  and  reprinting  times  could  barely  keep  up  with  the  changes, 
the  obligation  to  have  an  up-to-date  copy  of  each  POI  remained.  In  an 
effort  to  satisfy  this  requirement,  work  on  an  on-line  programming  routine 
(an  "editor")  to  effect  these  changes  was  commenced.  One  of  the  Chanute 
author  staff  was  given  the  task  of  creating  such  an  editor  which  would 
allow  POI's  to  be  changed  with  minimal  effort. 

A study  of  the  potential  cost  savings  of  the  POI  editor  was  included  as 
part  of  the  contract  for  which  this  report  is  being  produced  (CERL,  1975, 
p.  38;  also  Dallman,  1975b,  p.  3).  However,  after  the  form  of  the  contract 
was  finalized,  the  Chanute  development  effort  was  abandoned.  Under  normal 
conditions,  the  POI's  are  only  changed  on  the  average  of  once  a year,  and 
consequently  the  need  for  such  an  on-line  capability  would  be  limited. 

Though  it  was  decided  that  work  on  the  editor  be  suspended  before  a fully 
operational  routine  was  achieved,  it  should  be  noted  that  enough  work  had 
been  done  so  that  it  was  possible  for  this  evaluator  to  prepare  a working 
editor  after  some  minor  empirical  testing. 

Student  Mode 

Storage  of  student  answers.  An  expanded  version  of  a student  test 


data  collection  package  written  at  Chanute  includes  a listing  of  all 
responses  for  each  question  on  a test  (Figure  5) . According  to  Air  Force 


Figure  5:  Listing  of  Test  Item  Responses 
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regulations,  the  instructor  and  his  supervisor  are  required  to  give  a 
t written  explanation  for  any  question  on  a test  that  is  missed  by  all  the 

I 

students,  none  of  the  students,  or  answered  with  the  same  response  by  all 
the  students.  When  the  PLATO  system  is  used  for  collection  of  individual 
item  responses,  the  instructor  and  his  supervisor  have  all  the  information 
readily  available  to  make  a diagnosis  of  the  above  problems. 

Data  gathered  in  an  analysis  of  materials  produced  at  Chanute  also 
suggested  that  the  test  item  responses  were  in  fact  used  for  formative 
modifications:  that  is,  the  test  questions  were  revised  to  make  them 
easier  to  understand.  For  further  information,  see  the  section  on  "Tests", 
and  Appendix  II  in  Klecka  (1977). 

Storage  of  Instructor  comments.  The  recording  and  storage  of  the 
Instructor  comments  have  also  been  automated  (Figure  6).  After  noting  one 
or  more  incidents  requiring  analysis  and  commentary,  the  Instructor  can 
advance  an  explanation  for  the  actual  student  responses.  Then  the  super- 
visor, before  making  his  own  comments,  can  review  the  actual  question(s) 
along  with  the  observed  breakdown  of  the  responses.  These  remarks  are 
then  stored  and  are  made  accessible  to  both  instructors  and  other  super- 
visors. Both  sets  of  comments  can  be  used  to  modify  the  test  question(s) 
if  such  action  is  deemed  appropriate. 

All  the  data  necessary  for  making  a comment  are  readily  at  hand.  The 
comments  themselves  take  the  form  of  a brief  phrase  or  two  giving  a 
suggested  explanation  for  the  particular  phenomenon  that  occurred.  Much 


J 


time  is  saved  in  making  these  comments  since  the  individual  test  papers 
need  not  be  drawn  from  one  file  and  the  comments  selected  from  another  file. 
All  can  be  viewed  in  a few  minutes  without  the  instructor  leaving  the 
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Figure  6:  On-Line  File  of  Instructor/Supervisor  Comments 
i 

Class  number  / 760204 

j Instructor  / blake 

J Date  / 05/05/76 

I test  was  reviewed  and  considered 

valid.  60%  of  the 

i class  missed  question  20  seems 

> they  were  confused  with 

‘ the  source  of  the  oil 

flow  rather  than  the  path  of 
oil  flow,  test  was 
critiqued  in  class. 

Class  number  / 760204 
Supervisor  / rogers 
Date  / 05/05/76 

material  pertaining  to  question  20 
is  well  covered  in  plato 
lesson,  question  is  valid. 
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terminal.  The  savings  in  time  and  effort  of  limited  manpower  is  consider- 
able since  this  review  is  a mandatory  process,  but  it  is  not  easy  to  estimate 

1 

precisely  how  cost  effective  the  new  computerized  system  is.  Furthermore, 
the  review  process  has  been  improved,  in  the  opinion  of  the  Chanute  instruc- 
tor staff,  because  the  presence  of  both  data  and  comments  in  an  easily  ! 

accessible  format  encourages  and  facilitates  a more  thorough  evaluation. 

Retrieval  of  data  and  use.  According  to  Chanute  staff  who  were  inter- 
viewed, the  instructors  have  accepted  computerized  data  collection.  At  first, 
when  they  did  not  adequately  understand  the  sytem  or  perhaps  felt  their  jobs 
were  threatened  by  it,  the  instructors  apparently  were  not  very  enthusiastic. 

However,  after  they  saw  how  their  time  could  be  better  spent  on  individual-  ' 

ized  help  for  their  students  and  how  much  more  accurate  the  PLATO  IV  system 
could  be,  their  own  experience  convinced  them  of  its  effectiveness. 

The  reliability  of  the  old  manual  system  of  recording  data  is  not 

known  precisely.  However,  it  was  widely  regarded  by  the  Chanute  staff  as  a 

very  laborious  precedure:  the  results  of  block  tests  with  30  questions  each 

and  5 categories  of  possible  answers  ("a"  through  "e")  for  11  students  per 

class  had  to  be  transcribed.  Alternatively,  when  the  instructors  Interacted 

directly  with  the  computer,  the  necessary  procedures  for  utilizing  the  data 

collection  routines  were  seen  to  be  quite  straightforward  and  less  tedious 

2 

than  the  manual  method  of  inserting  numbers  into  a long  AF  form. 

On-line  lessons  were  also  used  to  maintain  an  up-dated  status  report 
on  courses  undergoing  revision  by  the  Instructional  System  Development  (ISD) 

2 

This  was  reported  by  PLATO  project  authors  ^^o  trained  the  Instructor 
staff  in  the  use  of  these  routines  and  also  provided  a special  lesson 
("chahelp")  as  an  on-line  aid  to  the  various  data  collection  procedures  that 
were  available. 
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group  (Figure  7).  Projected  dates  were  given  for  the  work,  or  the  course 
was  marked  as  completed  when  it  was  done.  Thus,  a continually  updated 
picture  of  the  status  of  the  course  could  be  maintained  and  easily  accessed. 
The  instructors  concerned  could  be  kept  abreast  of  any  developments  related 
to  their  courses. 

A pre-test  has  been  put  on-line  so  that  prospective  students  can  be 
ranked  approximately  according  to  their  probable  achievement  in  technical 
areas  ("cha93").  At  one  time  a paper  version  of  the  pre-test  was  used  for 
five  courses  (25-30  students  per  month)  in  the  Department  of  Weapons  System 
Support  Training  to  give  the  Instructors  an  idea  of  what  types  of  students 
would  comprise  each  new  class.  The  test  seemed  to  work  well  for  students 
in  the  vehicle  training  courses,  but  for  students  in  other  disciplines  such 
as  airframe  maintenance,  it  did  not  adequately  reflect  the  likelihood 
of  their  success. 

The  pre-test  shows  a definite  vehicular  orientation,  and  this  may  have 
been  a cause  of  its  limited  validity  when  used  for  other  courses.  The 
pre-test  is  now  administered  only  to  vehicle  maintenance  students.  Here 
too,  the  scores  are  stored  and  referenced  by  surname  while  the  students  are 
given  an  aptitude  level  for  the  following  areas:  general,  mathematics, 
vocabulary,  easy  and  hard  technical. 

In  all  these  areas  of  test  administration,  the  different  routines 
collect  and  store  the  data  until  the  results  are  needed.  In  addition,  the 
files  (i.e.,  storage  locations  on  computer's  magnetic  disk)  can  be  cleared 
out  to  make  room  for  new  lists  of  students  when  the  instructors  decide  to 
update  them.  The  procedures  for  entering  a new  class  and  reinitializing  the 
data  storage  bank  appear  to  be  simple  enough  tasks  so  that  the  instructors 


w 


r 


38 


Figure  7:  Status  of  Courses  under  Revision 

1 

J TWSV  05/07/76 
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3ABR47231A 
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6. 
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Comp 

480 

hrs 

7. 

3ABR47000C 

Comp 

360 

hrs 

8. 

3AZR47251B-1 

Jun 

76 

80 

hrs 

9. 

3AZR47252  -1 

Dec 

76 

240 

hrs 

10. 

3AZR47252  -2 

Dec 

76 

104 

hrs 

11. 

3AZR47252  -3 

Dec 

76 

120 

hrs 

12. 

3AZR47252  -4 

Dec 

76 

40 

hrs 

13. 

3AZR47252  -5 

Oct 

76 

40 

hrs 

14. 

4AST47250  -1 

Sep 

76 

80 

hrs 

15. 

4AST47250  -2 

Dec 

76 

0 

hrs 

Explanation 

ABR  = airman  basic  resident  (a  beginning 

course  like  the  GPV  and  SPV  courses) . 

ASR  = temporary  course  for  3,  5,  or  7 skill 
levels  (journeyman  or  advanced). 


AZR  = permanent  course  for  3,  5,  or  7 skill 
levels  (journeyman  or  advanced). 
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do  not  need  assistance  from  the  project  staff,  according  to  the  staff 

i 

i members  consulted  in  an  interview.  In  this  application,  the  computer 

allows  a new  service  to  be  provided  expeditiously.  Unfortunately,  no 
cost  savings  is  calculable.  Presumably,  the  effectiveness  of  the  instruc- 
tors is  enhanced  by  early  diagnosis  of  students'  potential  problems. 

* / 

' The  above  views  are  based  on  lengthy  interviews  with  the  Instructor  staff 

I at  Chanute. 
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Conclusions 

During  the  planning  stages  of  the  project,  Chanute  staff  found  only 
limited  management  applications  for  the  PLATO  system.  The  capabilities 
of  the  computer  became  more  significant  for  handling  the  substantial 
amount  of  data  collected  during  the  formative  development  period.  As  the 
project  matured,  the  computer  assumed  a larger  role  as  the  data-keeper  for 
managing  students  or  information  about  students. 

A precise  cost/benefit  analysis  is  difficult  because  many  computer 
applications  allowed  an  improved  quality  to  be  achieved  for  a task  which 
had  no  criteria.  For  example,  the  existence  of  readily  available  data 
collection  and  management  routines  permitted  and  encouraged  more  careful 
analyses  before  making  decisions.  Although  "better  decisions"  are 
certainly  a desirable  goal,  it  is  difficult  to  determine  a reasonable  cost 
for  them  in  concrete,  dollars-and-cents  terms.  In  other  cases,  there  is  no 
clear-cut  way  to  compare  the  accuracy  of  computer-tabulated  data  against 
previously  unaudited  entries  or  to  assign  an  economic  value  to  the 
(presumably)  more  accurate  data. 

In  any  event,  the  use  of  the  PLATO  system  at  Chanute  for  testing  and 
data  management  was  significant  for  the  following  reasons : 

1.  it  freed  the  instructors  from  tedious,  time-consuming 
tasks  to  give  more  attention  to  helping  individual 
students; 

2.  it  reportedly  improved  the  accuracy  in  recording 
specific  responses  to  questions  as  well  as  the  test 
scores  themselves; 

3.  it  allowed  rapid  and  convenient  access  to  relevant  data 
for  instructors  and  their  supervisors  who  had  need  of 
the  information; 


4.  it  made  retrieval  of  the  actual  test  items  as  well  as 
the  results  of  the  tests  readily  accessible  on-line. 
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Summary  Tables 

For  the  convenience  of  the  reader,  observations  related  to  the 
detailed  data  presented  in  the  tables  are  given  below. 

1.  The  number  of  lessons  used  for  data  collection  and 
testing  of  students  as  well  as  the  number  of  times 
they  are  accessed  indicates  such  use  was  of  substan- 
tial importance  to  the  operations  of  Chanute:  from 
October  8,  1975  until  October  1,  1976  these  programs 
were  used  for  over  800  hours. 

2.  The  amount  of  time  originally  expended  by  the  authors 
in  the  preparation  of  these  programs  is  difficult  to 
determine  since  exact  figures  on  production  rates 
were  not  kept;  however, 

3.  The  Chanute  staff  indicated  that  approximately  one 
month  was  spent  preparing  the  revised  forms  of  those 
programs  with  old  and  new  versions  described  in  the 
appendices. 

4.  Student  response  data  has  given  authors  guidance  for 
modifying  and  improving  tests. 

5.  The  Chanute  author  staff  is  continually  working  to 
improve  the  data  collection  routines  by  streamlining 
the  programs  to  enable  more  efficient  coding  and 
thereby  reduce  ECS  usage. 

6.  Efforts  to  improve  the  data  collection  routines  indi- 
cate that  these  programs  are  considered  useful  for 
the  instructors  and  students,  as  well  as  the  author 
staff. 
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Tables  of  Selected  Test  and  Data  Management  Files 


Introduction 

The  following  are  descriptions  of  PLATO  programs  used  for  the 
collection  of  student  responses  to  questions  and  scores  on  tests  at  Chanute 
as  well  as  sample  tests  themselves  and  other  data  collection  programs. 

Some  comments  on  them  are  given,  as  well  as  the  amount  of  space  (ECS) 
needed,  a factor  involved  in  their  utilization.  Furthermore,  in  several 
cases,  there  is  an  old  version  of  a program  that  has  been  or  is  being 

phased  out  by  a newer  improved  version  requiring  less  space.  Data  are 

given  for  both  versions  in  those  cases.  Readings  were  taken  on  September  15, 
1976  except  where  another  date  is  indicated. 

Explanation  of  Terms 

1.  High  ECS  = over  10,000  words 

A large  space  requirement  would  tend  to  make  it  diffi- 
cult to  use  the  lesson  between  10:00  a.m.  and  3:00  p.m. 
on  weekdays  when  the  system-wide  demands  for  ECS  are 
greatest. 

2.  Usage : number  of  times  lesson  has  been  accessed  and  the 

amount  of  time  spent  in  the  lesson  since  it  was  created 

or  since  October  8,  1975  when  accounts  data  keeping  was 
begun. 

3.  Non-Chanute : usage  of  lessons  at  other  sites  such  as 
Parkland  College  in  Champaign. 


j 

I I 
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Table  1:  Test  Item  Response  Collection 


Program: 

spshred668 

(formerly  "cha25") 

ECS  = 8065 

(formerly  11595) 

< 

Subject  matter: 

AF  form  668 

- item  analysis  for  test  evaluation. 

J Usage: 

Uses 

Hours 

Total 

3752 

502.8 

1 

Chanute 

3138 

464.0 

1 

Non-Chanute 

219 

38.8 

(Figures  on 

usage  as  of  September  20,  1976.) 

Program: 

cha95 

ECS  = 6392 

Subject  matter: 

MVE  results 

from  datafile 

: dally  appraisal  of  end- 

of  lesson  test  results. 

Usage: 

Uses 

Hours 

Total 

274 

16.2 

Chanute 

55 

2.2 

Non-Chanute 

219 

14.0 

Program: 

chal39 

ECS  = 7163 

Subject  matter: 

Block  and  pre-test  scores 

and  analysis  pre-  vs.  post 

test  scores 

• 

Options: 

1.  update 

file 

2.  student 

data  and  test 

data 

3.  block  averages 

Observation : 

The  figures 

for  "chal39" 

are  as  of  August  1,  1976. 

Deletion  of 

this  program 

occurred  after  that  date 

since  it  had  had  limited 

use,  according  to  Chanute 

staff  who  were  consulted 

on  September  17,  1976. 

Usage : 

Uses 

Hours 

Total 

165 

6.2 

Chanute 

4 

.1 

Non-Chanute 

161 

6. 1 

L A 
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Table  2;  Student  Management  and  Instructor  Help 


Program: 


charich 
ECS  = 1500 


(formerly  "cha39") 
(formerly  = 3252) 


Subject  matter:  Enrichment  lesson  list 


Observation: 


The  enrichment  lessons  were  used  when  a student 
completed  the  assigned  lesson  before  the  time  period 
was  up.  The  purpose,  according  to  one  staff  member, 
was  managing  students  who  might  otherwise  have  wan- 
dered around  the  room  and  interfered  with  other 
students  still  working. 


Usage : 


Program: 


Total 

Chanute 

Non-Chanute 


cha93 

ECS  = 5334 


Uses 

2244 

2194 

50 


Hours 

60.9 

57.2 

3.7 


Subject  matter:  Pretest  to  determine  student  aptitudes  in  various  areas. 

Options:  1.  all  students'  scores 

2.  weak  students'  scores  by  area 

3.  total  test  analysis 
■ 4.  individual  students 


Usage: 

Uses 

Hours 

Total 

719 

191.7 

Chanute 

474 

180.7 

Non-Chanute 

241 

11.0 

Program: 

Qha69 

ECS  = 1500 

Subject  matter: 

Special  Vehicles 

POI  Form 

with  : 

Observation: 


Examination  of  this  routine  revealed  that  no  special 
editor  was  needed. 
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Table  2 con't 


Program:  chahelp 

ECS  = 2914 

Subject  matter:  Orientation  to  PLATO  use  for  instructors. 

Options:  (table  of  contents  page) 

1.  introduction 

2.  writing  test  comments 

3.  entering  students  in  a course 

4.  deleting  students  from  a course 

5.  assigning  modules 

6.  lesson  sequencing 

7.  assigning  block  or  shred  tests 

8.  locating  student  statistics  (i.e.,  grades) 

9.  notefiles  (instructor  and  student) 

Usage  Uses  Uses  Hours 

Total  155  7.9  (all  by  Chanute  as  of 

October  20,  1976) 


Program:  chal32 

ECS  = 4147 

Subject  matter:  Data  on  courses  and  ISD  work. 

Options:  1.  data  by  branch 

2.  data  by  type  of  course 

3.  percent  completed 

4.  completed  courses 

5.  all  courses 

6.  add  new  courses 

7.  scheduled  courses 

8.  overdue  courses  and  courses  due  next  60  days 


Usage : 

Uses 

Hours 

Total 

344 

17.8 

Chanute 

191 

11.3 

Non-Chanute 

153 

6.5 

Program:  blocktOOl 

ECS  = 3250 


Course:  SPV  Subject:  block  x 

Subject  matter:  Sample  of  a typical  block  test  (there  were  approximately 
15  of  these  block  tests  on-line  as  of  November  3,  1976). 
Questions  on  a variety  of  topics  which  the  student  had 
to  learn  by  self-study  outside  the  classroom. 

Observation:  Very  easy  for  instructors/supervisors  to  read  other 

comments  and  write  their  ovm  in  this  lesson,  even  for 
someone  with  limited  experience  on  the  PLATO  system. 

Usage:  Uses  Hours 

Total  14  0.7  (all  by  Chanute) 


Program:  shred511a 

ECS  = 3746 

Subject  matter:  Block  test  for  common  course  segment  (this  is  a sample 
of  23  block  tests  now  on-line) . 

Observation:  Four  block  exams  for  the  portion  of  the  course  common 

to  the  four  shreds  (i.e. , the  portion  of  the  SPV  for 
which  the  PLATO  lesson  materials  were  developed)  are 
on-line  and  have  been  used.  Following  the  six  week 
common  course  segment,  the  students  were  given  instruc- 
tion in  their  own  specialty  in  each  of  the  four  shreds. 
Block  exams  during  this  specialized  training  period 
are  called  "shred  exams".  At  the  time  of  the  inter- 
view, all  six  of  the  "D"  shred  block  exams  were  on-line 
and  in  use.  They  had  been  put  up  gradually,  or  at 
least  not  simultaneously.  The  "B"  and  "C"  shreds 
each  had  two  exams  to  go  on-line  right  away  and  two 
more  coming  soon.  The  "A"  shred  had  not  been  revised 
by  ISD,  and  hence  its  block  exams  would  not  be  on-line. 


Usage : 


Uses 

81 


Total 


7.2  (all  by  Chanute) 
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Program: 


Options : 


Observation : 


Usage : 


Table  4:  Test 

Results  and  Analysis 

spgrades 

(formerly 

"cha90") 

ECS  = 2524 

(formerly 

= 3267) 

Block  test  results  (names. 

items  missed 

1.  see  data  by 

class 

2.  see  data  by 

test 

3.  see  data  by 

student 

4.  zero  data 

5.  'update 

The  above  options  show  how 

easy  it  was 

and  reinitialize 

the  recording  of  data, 

novice  Instructor. 

Uses  Hours 

Total 

496 

16.7 

Chanute 

276 

9.8 

Non-Chanute 

220 

6.9 
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Abstract 

A survey  of  the  process  of  lesson  revision  and  development  at 
Chanute  AFB  was  considered  important  to  determine  its  effects  on  the 
lessons  and  the  pedagogical  impact  of  the  computer-based  education 
materials.  Eight  representative  lessons  were  selected  by  a Chanute 
evaluator.  In  the  course  of  revision,  these  lessons  were  subjected  to 
a series  of  modifications  by  Chanute  staff  members  to  insure  their 
validation  when  used  by  military  students.  An  MTC  evaluator  prepared  an 
in-depth  analysis  of  the  "before"  and  "after"  versions  of  each  lesson 
to  ascertain  the  changes  and  their  impact.  Evaluators  at  Chanute  were 
given  draft  versions  of  the  report  for  their  comments  which  were  included 
in  the  final  version. 
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Introduction 


Background  and  Objectives 

This  is  a study  of  the  changes  made  between  two  series  of  Chanute 
lessons:  an  early  and  a later  series.  The  first  series  consisted  of 

lessons  before  validation  had  been  attempted,  but  after  trial  students 
had  been  run  and  after  initial  lesson  development  had  been  halted.  The 
second  series  included  these  same  lessons  after  they  had  undergone  revi- 
sion, where  necessary,  based  on  student  data.  The  revisions  were  made 
by  Chanute  staff  to  increase  the  likelihood  that  they  would  validate: 
i.e.,  that  90%  of  the  students  would  get  80%  or  better  on  the  end  of 
lesson  test. 

The  objectives  of  this  report  are  two-fold: 

1)  to  examine  the  lessons  to  ascertain  what  changes 
have  been  made,  and 

2)  to  determine  the  impact  of  the  changes  on  student 
performance. 

A more  thorough  discussion  of  the  individual  lessons  can  be  found  in 
An  Overview  of  Chanute  Lessons,  MTC  report  #10.  The  author  assumes  that 
the  reader  has  read  or  has  available  that  report  since  it  will  be  referred 
to  occasionally  (hereafter  called  the  Overview). 

Method  of  Approach 

These  analyses  compare  two  versions  of  eight  lessons:  one  version 
early  in  its  developmental  process  and  another  version  which  had  been 
modified  over  approximately  9 months,  including  6 months  use  by  a total 
of  about  150  students.  (See  Appendix  I for  details.)  The  ways  in  which 
each  version  differed  were  noted  and  evaluated.  By  this  procedure  some 


52 


insight  can  be  gained  into  the  revision  process  used  to  improve  the 
probability  of  validation  and  into  the  process  of  lesson  development 
as  practiced  in  the  Chanute  PLATO  project. 

' 

The  type  and  quantity  of  changes  were  by  no  means  uniform.  In 
some  cases  the  modifications  were  of  minor  importance,  perhaps  even 
negligible.  In  others  there  were  significant  structural  and  pedagogical 
remodelings.  As  might  be  expected,  the  latter  will  be  explored  in 
greater  detail  since  they  have  had  a greater  impace  on  the  achievement 
of  lesson  objectives.  Throughout  this  chapter,  the  lessons  will  be 
referred  to  by  a name  and  number,  e.g.,  "PT0"(1)  indicates  the  pre- 
validation version  while  "PT0"(2)  indicates  the  post-validation  version 
of  the  lesson  on  the  Power  Take-off  unit. 

As  part  of  the  formative  development  of  this  report,  draft  copies 
were  submitted  to  the  evaluators  and  authors  at  Chanute.  Additions  and 
corrections  were  included  in  the  report  at  appropriate  places.  In 
addition  the  Chanute  evaluator's  comments  are  included  at  the  end  of 
this  report. 
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Structural  Analysis 

General  Comments 

In  all  lessons  but  "PTO" (2) , the  structural  changes  did  not  warrant 
a new  detailed  diagram  (i.e. , flowchart).  These  diagrams  (not  included  with 
this  report)  were  used  extensively  for  this  and  other  Chanute  reports, 
notably  the  Overview.  For  more  details  on  this  and  other  subjects 
mentioned  here,  that  report  should  be  consulted. 

One  observation  is  in  order  at  this  point.  The  lessons  in  the 
second  series  share  an  indexed  linear  format  (except  for  "emission"(2) 
which  is  basically  non-lndexed) . The  modular  approach  is  a later  addition 
to  the  instructional  design  of  the  lessons.  In  most  cases,  the  lessons 
are  now  organized  in  such  a way  that  the  student  can  choose  when  he  is  to 
do  each  section.  This  is  an  important  move  toward  increasing  the 
individualization  of  the  lesson. 

Specific  Comments 

”PTo"(i)  and  (2).  Since  the  copy  of  "PT0"(1)  available  to  the 
reviewer  was  not  complete,  the  revised  form,  "PTO" (2),  was  analyzed  in  some 
detail.  More  extensive  remarks  may  be  made  about  this  lesson  than  about 
the  others  in  this  series  because  of  the  lack  of  previous  commentary. 

"Starter" (1)  and  (2).  The  lesson  "starter"(l)  was  also  available 
only  in  an  incomplete  form.  The  later  version,  "starter" (2) , while 
differing  in  many  aspects  from  its  predecessor,  showed  only  minor  struc- 
tural differences.  Consequently  these  did  not  warrant  a new  schematic; 
they  will  be  mentioned  in  the  discussion  of  organization  ("intra-lesson"). 

"Starter" (1)  had  only  a superstructure  which  was  not  filled  in,  and 
"PT0"(1)  displayed  the  beginnings  of  a whole  lesson;  these  did  not  possess 


General  Comments  - Organization 


Inter-lesson 

"By-pass  test"  option.  The  most  significant  organizational  change  in 
these  lessons  is  the  elimination,  in  all  but  one  of  the  lessons,  of  the 
option  to  take  the  end  of  lesson  test  immediately  upon  entering  the 
lesson.  This  "by-pass"  test  option  is  now  available  only  in  "diesel" (2), 
and  its  availability  may  be  an  oversight  rather  than  a deliberate  design 
feature.  Consequently  the  student  must  now  go  through  the  whole  lesson 

before  he  is  allowed  to  take  the  end  of  lesson  (f.e. , validation)  test. 

According  to  Chanute  staff,  this  change  was  .originally  made  to 

furnish  additional  validation  data.  It  was  allowed  to  remain  after 
validation  because  it  was  decided  that  there  was  no  advantage  to  saving 
the  time  of  the  good  student  in  the  group-paced  usage.  Also,  students 
selected  the  by-pass  option  repeatedly  to  avoid  taking  the  whole 
lesson;  if  they  did  not  pass,  they  were  cued  to  be  especially  attentive 
to  certain  portions  of  the  lesson  and  may  have  neglected  other  areas  as 
a consequence.  The  option  was  removed,  rather  than  re-coded  to  allow 
only  one  attempt  at  the  end  of  lesson  test.  Apparently  few  students 
passed  the  test  as  an  option.  Thus  it  was  felt  that  the  students' 
time  would  be  better  spent  in  going  through  the  lesson  rather  than 
making  repeated  attempts  at  the  by-pass  test. 

"Teaching  points".  A noteworthy  addition  to  each  lesson  is  the 
presence  of  a detailed  outline  available  to  an  author  (NOT  a student) 
from  the  page  stating  the  objective.  This  outline  is  much  more  detailed 
than  the  brief  one  found  in  the  index  to  the  lessons.  In  only  two 
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lessons  ("driveshaft"  and  "electricity")  are  directions  given  for 
accessing  the  teaching  points;  in  the  others  it  was  discovered  by  an 
examination  of  the  coding.  The  reason  Chanute  ISD  (instructional 
system  development)  staff  stated  for  not  allowing  access  of  these 
"points"  was  that  "the  student  will  eventually  see  the  information  in 
the  teaching  points  when  he  takes  the  lesson."  The  reviewer  feels  that 
the  usefulness  of  the  teaching  points  to  the  student  may  be  substantial 
and  should  at  least  be  investigated.  (See  Appendix  IV  and  "General 
comments:  Components"  of  the  overview.) 

Other  new  material.  In  some  cases  information  is  presented  in  the 
second  series  of  lessons  that  does  not  appear  in  the  earlier  versions. 
Details  of  such  occurrences  will  be  found  in  the  section  on  "new  material". 
The  information  presented  in  that  section  was  not  felt  to  be  significant 
enough  to  change  the  organization  of  the  lesson  in  any  substantial  way, 
and  thus  is  not  included  in  the  following  discussion  ("General  comments: 
Intra-lesson") . 

Intra-lesson 

A few  specifics  on  significant  organizational  changes: 

"starter" (i) 

The  large  help  section  of  " starter" (1)  has  been  broken  up 
into  convenient-sized  sub-sections.  These  are  listed  on 
an  index  page  and  can  now  be  chosen  at  will  by  the  students. 

They  can  go  directly  to  a specific  topic  and  need  not 
wander  around  in  one  large  section  as  before. 

The  individual  section  pre-tests  (i.e.,  section  by-pass 
tests)  have  become  end  of  section  tests,  and  there  is  no 
option  available  for  taking  them  in  lieu  of  going  through 
the  particular  section. 


Of  the  eight  lessons  examined,  only  "starter" (1)  contained 
such  section  pre-tests. 
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^ "transmission" (2) 

* The  index  page  is  more  useful:  it  now  appears  at  the 

^ beginning  of  the  lesson,  instead  of  the  end  as  in  "trans- 

mission"(l).  Also  the  students  have  a choice  as  to  when 
each  section  will  be  done. 

4 

»PTo"(2) 

* This  is  a very  short  lesson  on  transfer  cases,  sprag  units, 

and  power  take-off  units.  Each  subject  area  is  contained 

^ in  a separate  module  which  can  be  taken  at  any  time.  The 

i end  of  lesson  test  can  be  attempted  only  after  all  three 

, sections  have  been  completed. 

The  remaining  lessons  are  virtually  identical  to  the  earlier  versions 

as  far  as  their  organization  and  structure  are  concerned.  In  other 

sections  within  this  review,  specific  comments  will  be  made  where  the 

lessons  differ  from  their  predecessors. 
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Specific  Comments 

1 

i 

' Teaching  to  an  Objective 

In  all  cases  the  objective  has  remained  the  same:  to  pass  the  end  of 
lesson  test  with  more  than  75-80%  accuracy.^  In  "starter" (2) , this  objec- 
’ tive  is  stated,  although  it  was  not  in  "starter"(l) . The  accomplishing 

of  the  objective  is  based  solely  on  passing  an  end  of  lesson  test.  As  in 
the  Overview,  much  attention  will  be  given  to  the  end  of  lesson  test 

I 

and  to  the  factors  affecting  the  student's  successful  completion  of  it. 
Terminology 

The  Overview  provides  an  extensive  analysis  of  this  subject. 

As  far  as  the  current  report  is  concerned,  "transmission" (2)  is  the  only 
lesson  to  show  appreciable  changes  in  this  category.  (See  Appendix  VI 
for  details.)  Usually  the  context  in  which  the  key  terms/concepts  appear 
must  still  be  used  for  explanatory  purposes.  There  has  been  no  signifi- 
cant shift  in  emphasis  toward  rigorous  definition  of  terminology  in  the 
later  series. 

Interaction 

In  general  the  amount  and  type  of  interaction  is  very  similar  in 

i 

both  sets  of  lessons.  However  a few  questions  have  been  added  or  rephrased 
and  will  be  mentioned  because  they  relate  to  the  items  on  the  end  of 

I 

( 

I lesson  test.  Frequently  the  phrasing  change  or  addition  brings  the 

I question  more  closely  in  line  with  test  items  seen  later.  Comprehension 


However,  the  criterion  for  a lesson  that  was  not  part  of  the  group 
of  eight  examined  in  this  report.  Engine  Lubrication  System,  was  relaxed 
during  revision  from  7 of  8 (88%)  to  6 of  8 (75%) . 


i 
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of  the  questions  may  also  be  improved  by  this  remodeling.  Examples  of 
these  are  as  follows: 

"emission" (2) 

One  multiple  choice  question  has  been  added:  it  is  on 
a topic  related  to  the  subject  matter  stressed  in  a 
previous  question — air  flow  around  the  draft  tube  in  a 
non-positive  crankcase  ventilating  system. 

The  author  rephrased  a completed  question  concerning 
the  location  of  the  entrance  point  for  air  in  the 
positive-open  type  of  crankcase  ventilating  system. 

The  old  question,  which  the  rephrase  replaces,  was  very 
similar  to  a previous  question.  Thus  the  rephrased 
question  is  important  because  it  eliminates  a nearly 
redundant  question  and  instead  emphasizes  previously 
unstressed  knowledge. 

"transmission” (2) 


Four  questions  have  been  added:  three  of  these  cover 
topics  for  which  there  were  previously  no  questions  in 
the  body  of  the  lesson,  but  which  were  covered  in  the 
test.  The  other  addition  is  less  significant  because  the 
material  it  stresses  is  already  covered  by  another  question. 

"dr iveshaf t" (2) 

A single  question  has  been  reworded  so  that  it  now  relates 
to  material  in  one  of  the  test  items. 

"electricity" (2) 

One  question  has  been  changed  from  an  open-ended  fill-in 
("What  would  you  call  the  smallest  piece  that  would  still 
be  the  same  kind  of  matter?—  any  good  guess  will  do)"  to 
a rhetorical  question  ("What  is  a molecule?") . A discus- 
sion of  that  subject  area  then  follows. 

In  general  the  questions  added  or  remodeled  are  directly  related  to 

items  on  the  test  and  apparently  stress  Important  concepts. 

Visual  Presentation 


In  the  majority  of  cases  the  graphic  displays  and  other  aspects 
of  the  visual  presentation  have  remained  the  same  in  the  second  set  of 


lessons.  However  a few  changes  can  be  detected: 


"emission" (2) 


One  small  change  is  evident  in  that  two  small  phrases 
are  underlined  which  stress  an  important  component  of 
one  of  the  crankcase  ventilating  systems  that  is 
reviewed  on  the  end  of  lesson  test. 

"starter" (2) 


Frequently  the  text  describing  a slide  is  superimposed 
over  the  colored  portion  of  the  fiche,  making  it  diffi- 
cult to  read  the  text.  This  is  the  case  whether  the 
old  or  new  fiche  is  used.  Also  in  some  cases  it  is  very 
difficult  to  discern  what  the  slide  is  presenting: 
there  is  no  lettering  on  the  fiche  indicating  what  is 
being  seen.  Upon  reading  the  above  comments  in  June/ 
July  1976,  the  Chanute  staff  replied  that  they  felt  all 
such  problems  had  been  corrected  by  reprogramming  and 
by  production  of  a new  fiche.  However,  the  reviewer  re- 
checked the  lessons  and  found  many  of  the  problems 
still  in  existence. 

"diesel" C2) 


Replacement  of  a mediocre  and  tedious  graphic  at  the 
beginning  (i.e.,  something  like  a cross  between  a 
computer  and  a stock  report  ticker  tape,  perhaps  an 
attempt  at  humor?)  with  a slide  of  a diesel  engine. 

Even  the  author  realized  the  impact  of  the  original 
since  he  allowed  the  NEXT  key  to  break  through  the 
plotting. 

"electricity" (2) 

Important  terms/concepts  in  several  units  of  text  are 
stressed  by  the  use  of  all  capital  letters. 

Thus  these  changes  appear  to  be  perceptible  attempts  at  heightening  the 

visual  impact  of  the  CBE  presentation. 

Feedback 


Some  changes  in  feedback  can  be  noted  here: 

"emission" (2) 

In  one  unit  the  new  material  that  once  appeared  in  the  feed- 
back position,  although  not  properly  feedback,  now  appears  in 
a separate  unit/frame.  (More  details  on  this  placement  of 
new  material  in  section  on  "feedback"  in  the  Overview. ) 
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"starter"(2) 

Throughout  this  lesson  the  answer  is  given  in  the  feed- 
back after  three  incorrect  tries.  This  enables  the 
student  to  move  on  without  halting  his  progress  when 
he  is  unable  to  proceed  (although  this  technique  may 
encourage  a careless  attitude  in  the  student  and  thus 
has  some  hazards) . 


"transmission" (2) 

In  one  unit  new  material  is  taken  from  the  feedback 
position,  though  it  too  really  wasn't  feedback,  and  put 
into  a separate  unit/frame.  If  new  material  is  presented 
to  the  student  at  the  time  he  is  given  or  is  expecting 
feedback,  he  may  pass  over  it  lightly  and  thus  may  miss 
valuable  information.  This  seems  especially  likely  to 
happen  if  the  student  has  been  exposed  to  intimidating 
or  abusive  feedback  in  other  lessons. 


Tests 


The  end  of  lesson  tests  are  the  sole  criteria  for  accomplishing 
the  objective  in  these  lessons.  There  has  been  considerable  remodeling 


of  these  tests,  as  seen  in  the  comments  below: 


"emission" (2) 

Two  questions  on  the  test  have  been  rephrased  for  clarity. 
See  Appendix  III  for  more  Information  on  this  subject. 

"starter" (2) 

The  original  end  of  lesson  test  in  "starter" (1)  has  been 
made  the  end  of  lesson  review  containing  seven  multiple 
choice  questions.  The  new  test  is  the  familiar  ten- 
item  matching  drill.  From  a subject-matter  point  of  view, 
the  original  test  seemed  more  difficult  and  may  have  also 
been  a more  valid  test  of  the  objectives. 


"transmission" (2) 

The  scoring  procedure  should  be  mentioned  since  it 
differs  from  that  in  the  previous  version.  In  "trans- 
mission" (1)  the  correct  and  incorrect  answers  were  shown 
after  the  test  was  graded.  Now  there  is  no  individual 
scoring,  but  the  student  is  given  refresher  information 
on  questions  he  missed:  brief  paraphrases  of  previously 
seen  material  or  the  material  Itself  is  shown  again. 
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Then  the  student  Is  taken  back  to  the  index  page  with 
specific  sections  marked  that  must  be  completed  before 

he  can  retake  the  whole  test.  This  is  another  variation  ,i 

on  the  forced  review  technique,  which  may  be  effective 
as  a pedagogical  device. 

"diesel"(2)  ■ 

? 

At  several  places  in  the  text  there  are  strong  hints  i 

that  certain  items  should  be  remembered.  Hence  though  1 

the  test  itself  has  not  changed,  its  difficulty  has 
possibly  been  reduced.  For  more  on  this,  see  Appendix  II 
to  this  report. 

"drive  shaft" (2) 

Only  two  of  the  ten  questions  on  the  end  of  lesson  test 
are  the  same  in  "drive  shaft"(l).  The  interaction 
in  the  lesson  iiow  follows  more  closely  the  information 
requested  on  the  end  of  lesson  test,  although  the  content 
is  roughly  the  same. 

The  following  table  is  a survey  of  the  changes  made  in  the  relation- 
ship of  interaction  and  items  on  the  test  between  the  two  series  of 

i 

lessons.  The  percentage  indicates  the  number  of  test  items  that  have  been 
previously  covered  via  interaction  in  the  body  of  the  lesson.  For  example, 
in  the  case  of  "starter"(2) , there  is  at  least  one  paraphrase  question  in 
the  text  for  every  question  on  the  end  of  lesson  test.  It  was  not 
considered  necessary  to  make  a separate  listing  for  those  instances  where 
a question  on  the  end  of  lesson  test  was  anticipated  by  more  than  one 
question  in  the  body  of  the  lesson.  The  benefit  was  thought  to  be 
problematical. 

Related  observations.  In  the  reviewer's  opinion,  the  reading 
ability,  difficulty  of  questions,  length  of  lesson,  etc.  may  have  an 
important  effect  on  the  performance  of  the  students:  e.g.,  "driveshaft" C2) 
is  very  brief  (about  20  frames).  Thus  even  if  the  percentage  is  not 
high,  the  student  may  do  better  than  expected  if  the  same  material 
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Table  5 


Percentage  of 

Test  Items 

Covered  via 

Interaction 

Lesson 

Version(l) 

Change 

Version(2) 

emission 

40% 

+30% 

70% 

starter 

20% 

+80% 

100% 

transmission 

40% 

+50% 

90% 

diesel 

30% 

+20% 

50% 

hydraulics 

100% 

0% 

100% 

drive  shaft 

40% 

+30% 

70% 

PTO 

— 

— 

40%^ 

electricity 

85% 

0% 

85% 

This  includes  one  question  on  the  test  not  covered 
at  all  via  interaction  or  text.  Examination  of  the  test 
results  shows  that  the  performance  of  the  students  on  this 
question  was  about  the  same  as  their  performance  on  other 
test  questions! 

were  embedded  in  a longer  lesson  so  that  short-term  retention  would  not  be 
the  main  factor  in  performance  on  the  test. 

It  should  be  mentioned  that  this  is  not  presented  as  any  definitive 
judgement  on  lesson  quality.  It  is  likely  that  the  questions  on  the 
end  of  lesson  tests  were  modified  in  an  attempt  to  make  them  easier  so 
that  students  could  pass  the  tests  and  the  lessons  would  validate.  Also 
the  reviewer  has  no  way  of  determining  what  was  considered  prior  knowledge 
for  the  students  in  going  through  the  lesson.  Thus  some  terms  and/or 
concepts  may  be  familiar  to  the  class  and  may  not  have  been  stressed  due 


to  this  prior  exposure.  Consequently  the  students  could  answer  the  test 
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questions  even  without  their  being  pre-tested.  In  this  way,  factors 
outside  the  boundaries  of  the  lesson  may  play  an  important  role  in  the 
success  of  the  lesson  as  a pedagogical  whole. 


Coding 

Major  problems. 

"drive  shaft" (2) 

On  the  index  page,  if  a student  presses  any  key  other  than 
the  ones  allowed  (i.e.,  a,  b,  c,  d,  e,)  an  execution  error 
occurs:  the  student's  progress  through  the  lesson  is 

interrupted  and  he  is  summarily  deleted.  This  error  was 
NOT  present  in  "drive  shaft"(l)! 

Minor  problems.  Some  minor  problems  were  discovered  in  the  course  of 
preparing  the  individual  reviews.  In  general,  it  is  the  policy  in  this 
report  to  list  only  changes  that  have  been  made  from  the  previous  series. 
However  in  this  section  on  coding,  uncorrected  errors  are  significant 
since  they  may  affect  student  progress  through  the  lesson.  Thus  they  will 
be  noted  here  when  they  have  been  passed  on  unchanged  from  the  early 
version.  A more  extensive  description  of  coding  problems  in  general  in  the 
Chanute  lessons  is  available  in  the  Overview. 

Specific  examples. 

"emission" (2) 

There  is  only  partial  erasure  of  a question  when  forced 
review  is  done  for  an  incorrect  response.  This  is 
uncorrected  from  previous  version. 

Two  "Press  NEXTs"  appear  in  the  unit  in  which  the  percen- 
tage of  correct  answers  on  test  appears.  This  is  uncor- 
rected from  previous  version. 

An  omission  of  a critical  command  in  certain  contexts 
may  lead  to  an  inadvertent  return  to  the  beginning  of 
the  lesson  and  loss  of  about  15-20  minutes  work.  This 
is  uncorrected  from  previous  version. 
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"starter"(2) 


Because  of  some  faulty  coding,  part  of  the  section  on 
"Control  Circuits"  is  not  functional.  Due  to  the 
unfinished  condition  of  "starter" ( 1) , this  problem 
could  not  be  detected  in  the  early  version. 

"transmission" (2) 

Some  minor  non-erasure  and  overprinting  occurs,  especially 
in  the  feedback  position.  This  is  uncorrected  from 
previous  version. 

"diesel" (2) 


Some  minor  overprinting  of  lines  of  text  occurs.  In  one 
case  a slide  is  available  via  LAB  but  its  availability 
is  not  indicated  for  the  student.  This  is  uncorrected 
from  previous  version. 

"hydraulics" (2) 

The  forced  review  for  missing  a drill  item  now  works: 
it  was  optional  in  "hydraulics" (1)  due  to  a minor  coding 
problem.  In  the  same  drill  when  the  student  misses  an 
item  three  times,  the  correct  answer  appears,  but  it  is 
printed  over  a subsequent  item  in  the  drill.  Thus  it 
is  difficult  to  read  the  correct  answer.  This  is 
uncorrected  from  previous  version. 

"drive  shaft" (2) 

At  various  places  in  the  lesson,  there  is  a long  wait  for 
the  student  while  special  characters  are  loaded:  this 
impedes  the  progress  of  the  student.  There  are  a total  of 
six  possible  loadings  of  special  characters  which  must 
occur  before  the  whole  lesson  is  completed.  Humerous  (?) 
comments  (i.e.,  "PLATO  has  now  prepared  some  strokes.") 
attempt  to  fill  in  the  delay.  This  is  the  same  as  in 
previous  version. 

"PT0"(2) 

Due  to  the  unfinished  condition  of  "PT0"(1),  all  of  the 
following  are  new  problems. 

A frame  seemingly  designed  as  an  introduction  (as  deter- 
mined both  by  its  content  and  its  name — "intro")  is  in 
fact  invisible  to  all  students  except  those  who  fall  a 
particular  question.  Then  they  are  sent  back  to  this  unit 
as  part  of  a forced  review. 
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One  question  requests  the  student  to  name  two  kinds  of 
transfer  cases  but  the  judging  sequence  will  count  either 
half  of  the  answer  as  completely  correct.  A very  minor 
change,  indicated  in  the  print-out,  will  prevent  this  from 
occurring. 

A "PRESS  NEXT"  appears  at  the  bottom  of  the  page  but  a 
student  response  must  be  made  before  the  student  can 
move  on  from  this  place.  In  essence,  the  NEXT  key  is 
non-functional  at  this  point,  and  the  directions  can 
confuse  the  student. 

"electricity" (2) 

Pressing  LAB  at  one  point  takes  the  student  back  to  the 
beginning  of  the  test.  At  that  point,  the  directions  in 
large  letters,  "DO  NOT  PRESS  NEXT",  overprint  on  "Press 
-NEXT-"  in  normal  size  writing.  This  is  a new  problem. 

Summary  on  Code 

Thus  it  is  evident  that  some  problems,  most  not  of  a serious  nature, 
remain  in  these  lessons.  These  errors  may  prove  to  be  a nuisance  or  a 
source  of  confusion  to  students  working  through  the  lesson.  Of  the  minor 
problems,  none  would  require  extensive  reworking  of  the  code  to  correct. 
Indeed  some  problems,  such  as  the  overprinting  in  "electricity"(2)  above, 
had  been  corrected  as  of  September  1975,  when  Varian  copies  of  the  block 
directory  pages  were  made  for  these  lessons. 

As  part  of  the  overall  evaluation  of  Chanute  lessons,  the  MTC  staff 
has  monitored  the  use  of  these  lessons  at  Chanute  and  at  Parkland  College. 
We  were  somewhat  surprised  to  find  that  execution  errors  were  still 
occurring  after  lessons  had  been  used  by  several  hundred  students.  Under 
ordinary  circumstances,  20-50  students  should  discover  all  errors. 

Although  a number  of  the  problems  reported  by  MTC  staff  have  been  fixed, 
some  have  remained  and  continue  to  cause  execution  errors  for  other 
students.  Minor  problems  of  overprinting  can  be  excused  by  insufficient 


manpower  to  "polish"  the  lessons,  but  it  Is  more  difficult  to  explain 
away  more  serious  problems  that  Interrupt  progress  in  the  lesson. 

New  material 

A small  amount  of  new  material  has  been  added  to  these  lessons.  In 
most  cases  it  consists  of  "teaching  points,"  reworkings  of  old  questions, 
and  rephrasing  of  some  items  of  feedback.  These  have  already  been 
discussed  in  the  Overview  (sections  on  General  Comments:  Components, 
Specific  Comments:  Interaction  and  Feedback.)  In  some  cases,  whole  units 
of  new  information  have  been  added: 

"transmission" (2) 

A small  unit  containing  new  material  on  the  torque  converter 
and  fluid  coupling  has  been  inserted  here. 

"PT0"(2) 

Due  to  the  unfinished  state  of  "PT0"(1),  almost  all  the 
material  here  is  new. 
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The  various  categories  of  specific  comments  can  be  summarized  as 
follows: 

1)  The  by-pass  option  has  been  eliminated  in  all  but  one 
of  these  lessons — "transmission" (2) . 

2)  The  objectives  of  these  lessons  have  not  changed. 

3)  The  amount  of  interaction  specifically  related  to  the 
end  of  lesson  tests  has  increased. 

4)  With  certain  notable  exceptions,  coding  problems  have 
ceased  to  be  an  important  factor  in  these  later  lessons. 

5)  In  several  cases  new  material  has  been  removed  from 
the  feedback  position  and  put  into  new  separate  units. 

6)  All  lessons  are  fully  operational  and  have  their  full 
complement  of  complete  units. 

Based  on  the  way  that  these  eight  lessons  changed  from  their  pre- 
validation to  their  validated  form,  one  can  characterize  the  process  of 
revision  to  improve  validation  capability  as  follows: 

1)  Lessons  which  validated  quickly  were  not  appreciably 
changed;  typically  only  coding  errors  were  corrected. 

2)  Long  lessons  which  did  not  validate  quickly  were  divided 
into  2-4  pieces.  (This  information  is  based  in  part  on 
conversations  with  Chanute  staff  authors  and  data  not 
found  in  the  eight  lessons  examined  in  this  report.) 

3)  In  general,  the  objectives  or  criteria  were  not  changed. 

4)  By  looking  at  item  results  from  the  end  of  lesson  tests, 
questions  with  consistently  low  performance  were  iden- 
tified. In  some  cases,  these  items  or  the  whole  tests 
were  rewritten;  in  most  cases.  Information  to  aid  stu- 
dent performance  on  those  items  was  stressed  in  the  body 
of  the  lesson.  Emphasis  was  provided  by  giving  the  stu- 
dent questions  similar  to  those  on  the  end  of  lesson  test. 
When  an  item  was  already  covered  with  a question  in  the 
body  of  the  lesson,  the  feedback  warned  the  student  to 
remember  the  answer  for  the  test. 
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5)  When  the  student  was  having  difficulty  answering  the 
question  in  the  body  of  the  lesson,  correct  answers  and 
Important  terms  were  accented  with  various  special 
display  techniques. 

6)  Indices  were  provided  to  allow  the  student  self  reviews, 
and  some  lessons  were  reorganized,  possibly  to  Improve 
the  flow  of  the  lesson. 

7)  Minor  problems — such  as  overprinting  of  several  layers 
of  text,  partial  erasures,  or  non-functional  instruc- 
tions to  the  student — are  not  fixed  unless  they  signi- 
ficantly Interfered  with  the  student's  progress. 

8)  Execution  errors  still  occur  after  more  than  one  year 
of  student  use. 
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Observations  from  Chanute 


1)  Many  changes  were  imposed  to  attempt  standardization 
of  lessons  (written  prior  to  June  1974).  During  ISD 
development,  most  effort  was  expended  on  new  lessons; 
the  prepared  lessons  were  not  modified  until  valida- 
tion. 


2)  Many  changes  were  the  result  of  the  initial  use  of  poor 
programming  techniques. 

3)  From  the  Chanute  evaluator's  observations  of  the  effort 
and  others,  it  seems  implicit  to  the  instructional  pro- 
gramming philosophy  that  students  will  learn  in  spite 
of  the  techniques  used.  Consequently  the  importance  of 
validation  techniques — when  considered  as  a group — lay 
in  the  debugging  of  a lesson,  naking  it  usable  for  the 
majority  of  students.  Though  it  seems  like  heresy  to 
observe  cases  of 


a.  reduced  standards, 

b.  cuing  students  to  test  items, 

c.  dividing  lessons  to  facilitate  validation, 

d.  pretest  item  practice,  and 

e.  reduction  in  item  difficulty, 

a perjorative  impact  on  lesson  performance  outcomes^ 
was  not  noted.  Validation  techniques  are  arbitrary  in 
many  cases,  and  therefore  can  be  flexible,  for  the  end 
result  is  a viable  product. 


The  evaluator  is  not  speaking  of  the  end  of  lesson  test — obviously 
these  should  improve — rather  he  refers  to  the  block  exams,  instructor 
assessment,  field  evaluation,  etc. 
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emlssion( 1) 

= mtccl 

emlsslon(2) 

= mtcclO 

starter(l) 

= mtcc2 

starter(2) 

= mtccll 

transmission (1) 

= mtcc3 

transmission (2) 

= mtccl 2 

dlesel(l) 

= mtccA 

dlesel(2) 

= mtccl3 

hydraulics (1) 

= mtccS 

hydraulics (2) 

= mtccl4 

drive  shaft (1) 

= mtcc6 

drive  shaft (2) 

= mtcclS 

power  take  off 

= pro  • mtcc7 

PT0(2) 

= mtccl6 

electrlclty(l) 

* mtcc8 

electricity 

= mtcclS 
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Appendix  II:  Stressing  of  End  of  Lesson  Test  Items 


via  Interaction  In  "diesel” (2) 


The  following  table  gives  information  on  the  relationship  of 
interaction  to  test  items  in  one  particular  lesson.  This  lesson  was 
chosen  because  of  the  particular  way  in  which  test  items  are  stressed 
in  the  body  of  the  lesson.  Each  "X"  represents  one  occurrence  of  the 
designated  characteristic.  The  stressing  of  information  can  be  divided 
into  specific  categories: 

% = percentage  of  students  answering  question  correctly 

(79  cases  total) 

i/a(dlr)  = virtually  identical  questions  in  the  body  of  the  lesson 

i/a(indir)  = related  quest ion /answer  stressed  by  interaction 

text  = information  for  answer  provided  in  text,  but  not 

emphasized 

f/b  = answer  to  this  question  provided  in  the  feedback  to 

another  question 

blatant  1 = feedback  to  a question  gives  "Remember  this  for  test." 

It  appears  in  "emission"(2)  but  NOT  in  "emission" ( 1) ’. 

blatant  2 = strong  warning  in  text:  "Don't  forget  (this)"  and/or 

answer  set  off  from  normal  text  via  unusual  positioning 
or  special  characters.  It  also  appears  in  the  later 
version  of  "emission"  only. 
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% 

i/a(dlr) 

(i/a(indir) 

text 

f/b 

blatant  1 blatant  2 

* 1 

100 

X 

! 2 

1 

85 

X 

X 

X 

3 

99 

XX 

' 4 

94 

X 

X 

X 

5 

75 

X 

X 

6 

71 

X 

7 

90 

XX 

8 

89 

X 

9 

85 

X 

X 

X 

10 

85 

X 

Summary  of  and  conclusions  to  be  drawn  from  Appendix  II: 

1)  Extra  assistance  is  provided  in  the  form  of  strong 
hints  to  remember  certain  bits  of  data. 

2)  Some  of  the  answers  to  test  items  appear  at  oblique 
places  in  the  lesson  (i.e. , within  textual  passages  or 
in  related  questions) . 

3)  As  mentioned  above,  some  are  hinted  at  so  strongly 

it  is  as  if  they  were  stressed  before  the  student  sees 
them  on  the  final  test. 

4)  Two  of  the  quest  ions.  (#5  and  //6)  are  given  very  heavy 
stress  by  the  use  of  strong  hints:  thus  their  percentage 
correct  figures  may  have  been  inflated. 


There  are  two  tests:  a final  quiz  (no  longer  equal  to  a by-pass 


test)  and  an  end  of  lesson  test.  The  former  is  unchanged;  in  the  end  of 

lesson  test,  however,  there  is  a slight  rephrasing  of  two  questions  which 

I makes  for  clarity  and  less  confusion  for  the  student  looking  for  the 

( 

right  answer.  Comparison  of  the  two  may  help  to  show  what  has  been  done 

I 

i to  improve  the  precision  of  the  questions  asked.  First  the  questions  from 

i 

) '’emission"(l) : 

2.  Component  that  uses  d.  PCV 

engine  vacuum  to  draw 
gasses  from  crankcase 
Intake  manifold. 

3.  Vane-type  rotary  a.  air  pump 

In  #2,  the  phrasing  is  misleading:  it  is  actually  the  vacuum  that  operates 
the  pcv  valve;  the  valve  does  not  "use"  the  vacuum.  In  the  following 
question,  there  is  a problem:  another  possible  match,  "h.  air  injection," 
would  be  just  as  likely  a choice,  and  as  correct,  as  the  one  allowed, 

"a.  air  pump." 

The  following  are  the  corrected  versions  of  the  questions  as  well 

as  the  change  in  the  phrasing  of  the  correct  answers: 

. 2.  Valve  that  is  operated  d.  PCV  valve 

by  engine  vacuum  and 
draws  gasses  from  the 
crankcase  to  the  intake 
manifold. 

3.  ^Type  of  pump  used  on  air  a.  Vane  type 

injections  system.  rotary  pump 
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The  author  seems  to  be  aware  of  the  original  vagueness  in  the  questions 
in  "emission" ( 1)  and  has  taken  steps  to  correct  the  problem.  Also 
the  appearance  of  "valve"  and  "pump"  in  both  the  description  and  the 
desired  answer  is  a strong  prompt. 

Results . The  rephrasing  of  the  two  questions  and  the  addition  of 
a question,  as  mentioned  in  the  section  on  interaction,  increases  the 
total  of  test  items  previously  stressed  by  interaction  to  70%  in 
"emission" (2) . 
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Appendix  IV:  Teaching  Points 

The  following  Is  a table  listing  the  occurrences  or  absence  of  a 
section  called  "teaching  points"  that  appears  In  some  of  these  lessons. 
If  present,  the  section  Is  only  accessible  to  an  author,  NOT  a student, 
via  DATA  at  the  appropriate  place  In  the  lesson.  In  some  cases, 
directions  are  given  as  to  Its  presence.  In  others  It  must  be  discerned 
from  an  examination  of  the  printout. 


Lesson 

Present? 

Directions 

emlsslon(l) 

no 

— 

emlsslon(2) 

yes 

yes 

starter(l) 

no 

— 

starter(2) 

yes 

no 

transmlssion(l) 

no 

— 

transmission (2) 

yes 

no 

diesel(l) 

no 

— 

dlesel(2) 

yes 

yes 

hydraulics (1) 

no 

— 

hydraulics (2) 

yes 

no 

drive  shaft (1) 

no 

— 

drive  shaft (2) 

yes 

no 

PTO(l) 

no 

— 

PT0(2) 

yes 

no 

electricity(l) 

yes 

no 

electriclty(2) 

yes 

no 
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Appendix  V;  Lessons  Reviewed  for  this  Report  Available 
in  a "fixed"  and  "unfixed"  Form 


"Fixed"  and  "unfixed"  Lesson  Names 


Lesson 

Fixed  form 

Unfixed  form 

emission 

mtcc 10 

cha3  + some  material  from  cha63 

starter 

mtcc 11 

cha41 

transmission^ 

mtccl2 

cha73 

diesel 

mtccl3 

cha74  + end  of  lesson  test  in  chal3 

hydraulics 

mtccl4 

cha78  + some  material  in  cha80/84 

drive  shaft 

mtcc 15 

cha8£ 

power  take  off^ 

mtccl6 

cha86 

electricity 

mtcclS 

cha97  + other  lessons  listed  in  index 
as  sections:  cha37,  100,  and  43 

Lessons  in  which  both  forms  are  by  the  same  author. 

The  answers  to  the  questions  which  are  strongly  hinted  at  (see 
Appendix  II)  are  found  in  the  "cha"  lesson;  thus  comments  as  to  when  they 
were  inserted  and  how  well  students  performed  on  them  (stored  in  data 
records)  are  difficult  to  correlate  due  to  the  "non-fixed"  state  of  the 
lesson. 

Some  additional  comments  to  keep  in  mind: 

1)  the  sequence  of  movement  of  the  student  is  from  the 
first  "cha"  lesson  to  the  others  via  a -jumpout-; 

2)  all  the  "cha"  lessons  have  been  subjected  to  frequent 
editing  so  that  they  are  NOT  exact  equivalents  to 
their  mtcc  counterparts. 


79 


Conclusions : 


1)  "PTO"  was  changed  considerably  while  "hydraulics"  was 
relatively  unmodified. 


2)  "Starter"  and  "transmission"  reflected  extensive  changes 
while  "diesel"  and  "drive  shaft"  did  not. 


